Trear: Transformer-Based RGB-D Egocentric Action Recognition

In this article, we propose a transformer-based RGB-D egocentric action recognition framework, called Trear. It consists of two modules: 1) interframe attention encoder and 2) mutual-attentional fusion block. Instead using optical flow or recurrent units, adopt self-attention mechanism to model the temporal structure data from different modalities. Input frames are cropped randomly mitigate effect redundancy. Features each modality interacted through proposed block combined simple yet effective operation produce joint representation. Empirical experiments on large sets: THU-READ first-person hand action, one small set, wearable computer vision systems, have shown that method outperforms state-of-the-art results by margin.